Real-time virtual lidar sensor

ABSTRACT

A virtual LiDAR sensor system fora motor vehicle includes a plurality of camera modules and algorithms that generate a depth image, a RGB image, and a segmentation information image. The system is implemented with an algorithm that associates the backscattered signals with information from a color-reflectivity table, incident angle determination and depth information.

INTRODUCTION

The present disclosure relates to a sensor for motor vehicles. Morespecifically, the present disclosure relates to a virtual LiDAR sensorfor motor vehicles.

Certain motor vehicles are equipped with cameras and sensors to evaluatethe surroundings of the motor vehicle. Moreover, various motor vehiclesemploy cameras and sensors when cruise control is activated. Particularsensors utilize LiDAR technology that creates a spatial point of themotor vehicle surroundings.

While current systems employing cameras and sensors to provide cruisecontrol and driver assistance achieve their intended purpose, there is aneed for a new and improved system to evaluate the surroundings to themotor vehicle.

SUMMARY

According to several aspects, a virtual LiDAR sensor system for a motorvehicle includes a plurality of camera modules and algorithm modulesthat generate a depth image, a RGB image, and an optional segmentationinformation image. The algorithms associate the backscattered signalswith information from a color-reflectivity table, incident light angledetermination and depth information.

In additional aspect of the present disclosure, the three images arecaptured by a respective camera module, algorithm module, or both.

In another aspect of the present disclosure, the three images areimage-pixels that are converted to a 3-d point cloud distribution.

In another aspect of the present disclosure, the system converts the 3-dpoint cloud into an intensity point cloud.

In another aspect of the present disclosure, the intensity point cloudincludes the 3-d point cloud and intensity information.

In another aspect of the present disclosure, the system includes a 3-dprojection module to generate the point cloud.

In another aspect of the present disclosure, information from the 3-dprojection module is transformed to 3-d coordinates.

In another aspect of the present disclosure, the 3-d coordinates areutilized in an incident angle module that provides incident angle forall individual points in the 3-d point cloud.

According to several aspects, a virtual LiDAR sensor system for a motorvehicle includes a first camera or a first camera system that capturesdepth of field information of an image, a second camera that capturesRGB information of the image, and a third camera or algorithm modulethat generates semantic segmentation information of the image. Thesystem is implemented with an algorithm that generates a depth andintensity point cloud of the image from the depth of field information,the RGB information and the semantic segmentation information.

In another aspect of the present disclosure, the system includes a 3-dprojection module that provides a pattern transformation.

In another aspect of the present disclosure, the pattern transformationincludes 3-d coordinates.

In another aspect of the present disclosure, the 3-d coordinates areassociated with propagation of a laser beam and backscatteringattenuation.

In another aspect of the present disclosure, the 3-d coordinates aretransmitted to an incident angle module.

In another aspect of the present disclosure, the incident angle moduleprovides information to a digital reflector.

In another aspect of the present disclosure, the RGB camera providescolor information to a color-reflectivity table.

According to several aspects, a virtual LiDAR sensor system for a motorvehicle includes a first module that provides depth of field informationof an image, a second module that provides RGB information of the image,a third module that provides semantic segmentation information of theimage, a geometry point cloud that receives information from the firstmodule, and a physics module that receives information from the geometrypoint cloud module and the second and the third modules. The physicmodule generates a depth and intensity point cloud.

In another aspect of the present disclosure, the physics module includesfour sub-modules.

In another aspect of the present disclosure, the four sub-modulesinclude a laser beam propagation sub-module, a backscatteringattenuation sub-module, a digital reflector sub-module and acolor-reflectivity table sub-module.

Further areas of applicability will become apparent from the descriptionprovided herein. It should be understood that the description andspecific examples are intended for purposes of illustration only and arenot intended to limit the scope of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are for illustration purposes only and arenot intended to limit the scope of the present disclosure in any way.The patent or application contains at least one drawing executed incolor. Copies of this patent or patent application with color drawingswill be provided by the U.S. Patent and Trademark Office upon requestand payment of the necessary fee.

FIG. 1 is a schematic of a LiDAR sensor system;

FIG. 2 is a block diagram of the overall implementation of a virtualLiDAR system according to an exemplary embodiment;

FIGS. 3A and 3B show a comparison between conventional LiDAR and theLiDAR system shown in FIG. 2 ;

FIG. 4 is a block diagram of the detailed implementation of the systemshown in FIG. 2 ; and

FIG. 5 is a block diagram describing the virtual LiDAR system shown inFIG. 2 for in-vehicle perception.

DETAILED DESCRIPTION

The following description is merely exemplary in nature and is notintended to limit the present disclosure, application, or uses.

Referring to FIG. 1 , there is shown a typical LiDAR system 10 for amotor vehicle. The LiDAR system 10 includes an emitter 12 that transmitsa laser beam 14 towards an object 18 and a reflector 16. The reflector16 provides backscattering attenuation to a receiver 20. In variousimplementations, the reflector 16 is a Lampert reflector.

Referring further to FIG. 2 , there is shown a virtual LiDAR system 30for a motor vehicle, in contrast to the system 10 shown in FIG. 1 , inaccordance with the principles of the present disclosure. The system 30includes a set of modules to generate an intensity point cloud (x,y,x,i)42. The set of modules 30 includes a first camera module 36 thatcaptures a depth of field of an image of an object; a second cameramodule 38 that captures RGB information of the image; and a third cameramodule 40 that provides semantic segmentation information of the image.The information from the camera modules 36, 38 and 40 is transmitted toa conversion module 32.

The conversion module 32 converts the image-pixel information form thecamera modules 36, 38 and 40 to a 3-d (x,y,x) point cloud. Theconversion module 32 further determines the incident angle of the laserbeam 14 upon the reflector 16, color information, pattern transformationand noise cancellation of the image.

Information provided by the module 32 is transmitted to a set of physicsmodules 34, including a laser beam propagation sub-module 44, abidirectional reflectance distribution function (BRDF) Lampert reflectormodule 46, a color reflectivity module 48 and a backscatteringattenuation module 50. The set of physics modules 34 then generate theintensity point cloud 42. The intensity point cloud (x,y,z,i) 42includes spatial information (x,y,z) as well as the intensity of theimage, which is indicated by RGB color variations shown in the intensitypoint cloud (x,y,z,i). As an example, high intensity is indicated in redand low intensity is indicated in blue. For the comparative purposes,FIG. 3A shows a 3-d point cloud 50 with spatial features (x,y,z) andFIG. 3B shows the 3-d point cloud along with intensity information toprovide an intensity point cloud 52 with spatial and intensity features(x,y,z,i).

In various arrangements, the camera modules 36, 38 and 40, theconversion module 32 and the physics module 34 are controlled by analgorithm implanted in an electronic control unit (ECU) situated, forexample, in a motor vehicle. The ECU is a non-generalized, electroniccontrol device having a preprogrammed digital computer or processor,memory or non-transitory computer readable medium used to store datasuch as control logic, software applications, instructions, computercode, data, lookup tables, etc., and a transceiver. Computer readablemedium includes any type of medium capable of being accessed by acomputer, such as read only memory (ROM), random access memory (RAM), ahard disk drive, a compact disc (CD), a digital video disc (DVD), or anyother type of memory. A “non-transitory” computer readable mediumexcludes wired, wireless, optical, or other communication links thattransport transitory electrical or other signals. A non-transitorycomputer readable medium includes media where data can be permanentlystored and media where data can be stored and later overwritten, such asa rewritable optical disc or an erasable memory device. Computer codeincludes any type of program code, including source code, object code,and executable code. The processor is configured to execute the code orinstructions.

The algorithm in various arrangements is an application implemented as asoftware program configured to perform a specific function or set offunctions. The application may include one or more computer programs,software components, sets of instructions, procedures, functions,objects, classes, instances, related data, or a portion thereof adaptedfor implementation in a suitable computer readable program code. Theapplications may be stored within the memory or in additional orseparate memory.

Referring to FIG. 4 , there is shown a modeling/simulation arrangement100 of the system 10 (FIG. 1 ) and modules 30 (FIG. 2 ). The arrangement100 includes a simulator 102 that transmits information to aconfiguration module 104. The configuration module 104 configures thecamera modules 36, 38 and 40 and the LiDAR attributes, such as, forexample, resolution, AOV and fps.

Information from the configuration module 104 and the depth of fieldcamera module 36 is relayed to a 3-d projection module 106. Output fromthe 3-d projection module 106 is transmitted to a pattern transformationmodule 108, which provides a spatial (x,y,z) point cloud 110. Anincident angle module 112 determines the angle of incident of the laserbeam on the BRDF Lambert reflector. Information from the conversionmodule is transmitted to the laser beam propagation sub-module 44, thebackscattering attenuation sub-module 50 and the BRDF Lambert reflectorsub-module 46 of the physics module 34. Color information from the RGBcamera module 38 is transmitted to the color-reflectivity tablesub-module 48. Labeling and noise cancellation information from thesegmentation camera module 40 is transmitted to the BRDF Lambertreflector sub-module 46 and color-reflectivity table sub-module 48.

If the intensity information from the physics module 34 exceeds a signalto noise (S/N) threshold, this information is utilized to generate theintensity point cloud 42 with spatial and intensity information. Invarious arrangements, information associated with the intensity pointcloud 42 is utilized in an interface with other software architecture114.

Turning now to FIG. 5 , there is shown the system 10 (FIG. 1 ) andmodules 30 (FIG. 2 ) implemented as an in-vehicle perception arrangement200. The arrangement 200 includes a module 202 with stereo cameras thatprovide a depth estimation. Alternatively, the arrangement 200 alsoincludes a module 204 with depth and/or regular photograph (RGB)cameras. Information from either module 202 or module 204 is relayed toa module 206 that generates a depth map and a module 208 that generatesa RGB image.

Geometric information (x,y,z) of the depth map is relayed from themodule 206 to a module 210. Further, the physics-based algorithmdescribed earlier utilizes information from the modules 206 and 208 toprovide an intensity estimation (x,y,z,i) to the module 210. The module210 then generates a 3-d point cloud with spatial information andintensity information. The 3-d point cloud generated in the module 210is utilized in a 3-d objection detection module 212, and the objectsdetected in the 3-d detection module are sent to a detection resultsmodule 214.

A virtual LiDAR system of the present disclosure offers severaladvantages. These include utilizing time-of-flight attributes andsimplified laser physics. Further, the virtual generation of a LiDARpoint cloud with intensity information achieves enhanced 3-d perceptionperformance.

The description of the present disclosure is merely exemplary in natureand variations that do not depart from the gist of the presentdisclosure are intended to be within the scope of the presentdisclosure. Such variations are not to be regarded as a departure fromthe spirit and scope of the present disclosure.

What is claimed is:
 1. A virtual LiDAR sensor system for a motorvehicle, the system comprising: a plurality of camera modules andalgorithm modules that generate a depth image, a RGB image and asegmentation information image; wherein the algorithms associate thebackscattered signals with information from a color-reflectivity table,incident angle determination and depth information.
 2. The system ofclaim 1, wherein the three images are captured by a respective cameramodule, algorithm module, or both.
 3. The system of claim 2, wherein thethree images are image-pixels that are converted to a 3-d point clouddistribution.
 4. The system of claim 3, wherein the system converts the3-d geometric point cloud into an intensity point cloud.
 5. The systemof claim 4, wherein the intensity point cloud includes the 3-d pointcloud and intensity information.
 6. The system of claim 1, wherein thesystem includes a 3-d projection module to generate the point cloud. 7.The system of claim 6, wherein information from the 3-d projectionmodule is transformed to 3-d coordinates.
 8. The system of claim 7,wherein the 3-d coordinates are utilized in an incident angle modulethat provides incident angle for all individual points in the 3-d pointcloud.
 9. A virtual LiDAR sensor system for a motor vehicle, the systemcomprising: a first camera or a first camera system that captures depthof field information of an image; a second camera that captures RGBinformation of the image; and a third camera or algorithm module thatgenerates semantic segmentation information of the image, wherein thesystem is implemented with an algorithm that generates a depth andintensity point cloud of the image from the depth of field information,the RGB information and the semantic segmentation information.
 10. Thesystem of claim 9, wherein the system includes a 3-d projection modulethat provides a pattern transformation.
 11. The system of claim 10,wherein the pattern transformation includes 3-d coordinates.
 12. Thesystem of claim 11, wherein the 3-d coordinates are associated withpropagation of a laser beam and backscattering attenuation.
 13. Thesystem of claim 11, wherein the 3-d coordinates are transmitted to anincident angle module.
 14. The system of claim 13, wherein the incidentangle module provides information to a digital reflector.
 15. The systemof claim 9, wherein the RGB camera provides color information to acolor-surface reflectivity table.
 16. A virtual LiDAR sensor system fora motor vehicle, the system comprising: a first module that providesdepth of field information of an image; a second module that providesRGB information of the image; a third module that provides semanticsegmentation information of the image; a geometry point cloud thatreceives information from the first module; and a physics module thatreceives information from the geometry point cloud module and the secondand the third modules, wherein the physic module generates a depth andintensity point cloud.
 17. The system of claim 16, wherein the physicsmodule includes four sub-modules.
 18. The system of claim 17, whereinthe four sub-modules include a laser beam propagation sub-module, abackscattering attenuation sub-module, a digital reflector sub-moduleand a color-reflectivity table sub-module.